We introduce Memory-QA, a novel real-world task that involves answering recall questions about visual content from previously stored multimodal memories. This task poses unique challenges, including the creation of task-oriented memories, the effective utilization of temporal and location information within memories, and the ability to draw upon multiple memories to answer a recall question. To address these challenges, we propose a comprehensive pipeline, Pensieve, integrating memory-specific augmentation, time- and location-aware multi-signal retrieval, and multi-memory QA fine-tuning. We created a multimodal benchmark to illustrate various real challenges in this task, and show the superior performance of Pensieve over state-of-the-art solutions (up to 14% on QA accuracy).

Many theories of discourse structure rely on the idea that the segments comprising the discourse are linked through inferred relations such as causality and temporal contiguity. These theories suggest that the resulting discourse is represented hierarchically. Two experiments examine some of the implications of these hierarchical structures on the perceived coherence of texts. Experiment 1 shows that texts with more levels to their hierarchical structure are judged to be more coherent. Experiment 2 demonstrates that these effects are sensitive to the genre of the text. Specifically, narratives seem to be more affected by manipulation of the discourse structure than procedural texts.